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This study explored the relationships between student achievement and student, 
school and home variables from the pan-Canadian assessment program administered 
by the Council of Ministers of Education Canada (CMEC): the School Achievement 
Indicators Program (SAIP) Mathematics 2001. The study also evaluated the datasets 
used in relationship to their utility for statistical modeling of school performance. 
Student beliefs about mathematics are positively related to mathematics achievement 
for both age groups and for both domains of mathematics. As students' use of 
instructional supports (parental assistance with mathematics homework, computers 
in the mathematics classroom) increases, there is an associated decrease in 
mathematics scores. 
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Dans cet article, les auteurs explorent les relations entre le rendement scolaire et 
certaines caracteristiques des eleves, des ecoles et de Tenvironnement familial a partir 
des donnees du programme d' evaluation pancanadien du Conseil des ministres de 
I'Education (Canada) (CMEC), le Programme d'indicateurs du rendement scolaire 
(PIRS) 2001 en mathematiques. L'etude evalue egalement les bases de donnees au 
regard de leur pertinence pour la modelisation statistique de la performance scolaire. 
Les auteurs notent une correlation directe entre les idees que se font les eleves au sujet 
des mathematiques et le rendement en mathematiques tant pour les groupes d'age 
que pour les domaines des mathematiques. Plus les eleves font appel a du soutien 
pedagogique (aide des parents pour les devoirs de mathematiques, ordinateurs dans 
les classes de mathematiques), plus les notes en mathematiques diminuent. 

Mots cles : rendement des eleves, performance scolaire, modelisation a multiples 
niveaux. 
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Students' performance in public education is an important concern for 
several reasons, fwo of which cenfre on fhe large scale of expendifures of 
public funds on educafion and fhe beneficial effecfs educafion has on fhe 
well being of bofh individual sfudenfs and sociefy (Hanushek, 2002). A 
key indicafor of educafional performance is fhe achievemenf of sfudenfs 
on cenfrally adminisfered fesfs of learning ouf comes (Haffie, Jaeger & 
Bond; 1999; Keeves, 2002; Shore, Madaus & Clarke, 2000). Testing 
sfudenf achievemenf af provincial, national, and infernafional levels has 
become a common mandafed practice fhroughouf Canada. 
Implemenfafions of large-scale assessmenf programs in Canadian 
schools are fypically predicafed on fhe assumpfion fhaf they will 
influence fhe qualify of educafion. For example, Onfario has esfablished 
an independenf fesfing organization fo conducf provincial assessmenfs 
wifh fhe express purpose "fo promofe greafer accounfabilify in Onfario's 
publicly funded educafion sysfem and fo improve fhe qualify of 
educafion" (Educafion Qualify and Accounfabilify Office, 2002, p. 1). 
However, fhe ways in which fesfing and fesf resulfs are related to 
system-level characteristics, such as educational quality, are not well 
understood. Factors in addition to curriculum and instruction, such as 
student characteristics on entry, teacher and school traits, and home and 
community characteristics, have been shown to be significantly related to 
student achievement scores. These relationships vary from one grade fo 
anofher, one school fo anofher, and one subjecf area fo anofher (Fifz- 
Gibbon, 1998; Ma, 2001; Mandeville & Anderson 1987; Rumberger, 1995). 

In spife of the apparent complexities associated with schooling and 
learning outcomes, the communications emanating from large-scale 
assessmenf programs fake fhe form of reporfing school and provincial 
averages of simple fesf scores or fhe percenfages of sfudenfs classified in 
differenf ordered achievemenf cafegories. Typically, fhese reporfs fail fo 
fake info accounf fhe confexfual relationships of fhese scores fo sfudenf, 
feacher, school, home, or communify fraifs. Instead, the reported results 
simply show that on average some schools perform better fhan ofhers in 
fhe differenf skill areas and af differenf grades. In some public reporfs 
(e.g., Cowley, Easfon, & Walker, 2001) schools are ranked in ferms of 
sfudenf resulfs on fhese tesfs (often by aggregafing resulfs across subjecf 
areas) fo describe and monitor sysfem qualify. The publicafion of fhese 
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rankings implies that variation in student performance is solely due to 
school effects. 

Because most information related to human endeavours is complex, 
the underlying context of factors and sifuafions fhaf influence fhese 
resulfs musf be faken info accounf fo undersfand educafional 
performance af fhe class and school levels (Join! Advisory Committee, 
1993; Sfafisfics Canada & Council of Ministers of Educafion Canada 
[CMEC], 2000). Raudenbush and Willms (1991) poinf ouf fhaf, in 
addition fo sfudenf variables such as gender, affifudes, and perceptions, 
class level variables such as composition and size, diversify of 
insfrucfional pracfice and feacher experience, as well as school variables 
such as climate, size, and academic focus should be considered. Willms 
and Kerckhoff (1995) suggesf fhaf fhe resulfs of large-scale assessmenfs 
have fo be reporfed and considered af fhree levels before resulfs are 
inferprefed: fhe gross level of fesf score averages; fhe level of nef 
producfivify in which scores are modified fo better reflecf fhe variation 
in fhe conditions of schooling; fhe level of inequality of sfudenf 
characteristics on enfry (see also Join! Advisory Committee, 1993, 
Guideline 2, Parf IV, Section B, p. 20). Much research has been conducfed 
on international and national assessment datasets yielding a wealth of 
informafion abouf patterns of sfudenf and school char acf eristics: fhe 
Programme of International Student Assessment (Fuchs & Woessmarm, 2004; 
Willms, 2004), fhe Primer Estudio Internacional Comparitivo of Latin 
American nations (Willms & Somers, 2001), fhe US National Education 
Longitudinal Study (Dumais, 2002), fhe US National Longitudinal Survey of 
Adolescent Health (Waff, 2003), sfafe level dafasefs in Ausfralia (Elill & 
Crevol, 1999) and fhe Unifed Sfafes (Reeves & Bylund, 2005). Elowever, 
fhe national assessmenf program in Canada, alfhough well over a decade 
old, has nof been well explored and analyzed. The purpose of fhis sfudy 
is fo explore a pan-Canadian assessmenf dafasef fo model sfudenf and 
school fraifs in relafion fo achievemenf and in doing so evaluafe fhe 
qualify and utility of fhese dafa for fhe analyses and modeling of school 
performance. 

The Correlates of Learning Oufcomes research program (COLO)' is a 
long-ferm research program designed fo use educafional indicafors fo 
elucidafe fhe underlying sfrucfures and processes of public educafion 
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systems in Canada. The results and findings of fhis research program are 
of inferesf and ufilify fo policy makers and educafors. The logic of fhis 
inferesf lies in whaf may be fermed fhe pafh of policy influence 
(Kennedy, 1999). The elemenfs of public schooling fhaf are accessible fo 
policy makers (such as funding, curriculum, fhe nafure and exfenf of 
insfrucfional supporf and supervision, fhe provision of opporfunifies for 
professional developmenf, and school organizafional sfrucfure) should 
show some influence over fhe key consequences of schooling (such as 
sfudenf learning, graduafion rafes, or ease of enfry info fhe labour 
market). The investigation and understanding of fhese pafhways fifs 
wifhin fhe realm of evidence-based research and decision making 
(Raudenbush, 2002; Slavin, 2002) and so, wifhin fhe scope of fhe sfudy 
reporfed here. 

In Canada, indicafors of sfudenf and school performance and 
characferisfics of sfudenfs and schools have been collecfed as parf of 
infemafional, nafional, and provincial assessmenf programs. These 
programs have yielded large dafasefs confaining measures of 
achievemenf of significanf learning oufcomes in fhe areas of reading, 
writing, numeracy, and science, as well as indices of sfudenf, home, and 
school characferisfics or correlates. Although for Canadian schools 
where a subsfanfial amounf of fhe score variance can be accounfed for by 
fhe classroom, fhe majority of fhese assessmenf programs do nof permif 
fhe analysis of dafa af fhe class level (Rogers, Anderson, Klinger, 
Dawber, Feng, & Zhou, 2006). Consequenfly, fhe resulfs reporfed are 
resfricfed fo fhe sfudenf and school levels. 

The sfudy reporfed here is based upon fhe pan-Canadian assessmenf 
program adminisfered by fhe Council of Minisfers of Education Canada 
(CMEC): fhe School Achievemenf Indicafors Program (SAIP). The 
assessmenf examined is fhaf for mafhemafics conducfed in 2001 (see 
CMEC, 2005 for full documenfafion) in which mafhemafics was divided 
info fwo achievemenf domains - Confenf and Problem Solving. The 
Confenf domain included numbers and operations, algebra and 
functions, measurement and geometry, and data management and 
statistics. Problem Solving included a range of problems and solutions in 
fhe areas of numbers and symbols, abilify fo reason and consfrucf proofs, 
providing information and making inferences, pursuing evaluafion 
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strategies, and demonstrating communication skills (CMEC, 2001). The 
study also evaluated the use of the SAIP data for analyses of fhis fype. 

The achievemenf fesfs were adminisfered fo sfrafified samples of 
sfudenfs, aged 13 and 16, wifhin each province and ferrifory fo allow 
reporfing af fhe provincial and nafional levels for each of fhese fwo age 
groups. In addifion fo fhe achievemenf fesfs, fhe sfudenfs, feachers, and 
principals in fhe sampled schools complefed quesfionnaires generafing 
data on student, class, school, and home variables. 

A total of 43,314 sfudenfs were in fhe dafasef (Table 1) wifh 1,746 
sfudenfs having no index of which mafhemafics fesf fhey complefed (4% 
loss). This dafa loss was nof considered sysfemafic because if was spread 
across age groups and mafhemafics domains, and fhe level of loss was 
considered low enough fo supporf fhe feasibility of furfher analyses. 
Issues of dafa qualify, such as missing dafa, dafa infegrify, and levels of 
dafa organizafion, were encounfered and are described separafely in 
anofher paper in fhis issue (Rogers ef al., 2006). 

Table 1: Number of Students in SAIP Mathematics 2001 Dataset 

Age 

Mathematics Domain 



13 years old 

16 years old 

Total 

CONTENT 

12 334 

8 842 

21 176 

PROBLEM 

11851 

8 542 

20 393 

Missing 

1052 

692 

1 746 

TOTAL 

25 237 

18 077 

43 314 


THE ANALYSES 

The data from fhe SAIP 2001 assessmenf of mafhemafics were divided 
info four achievemenf/age groups for analysis purposes: Problem solving 
- age 13, Problem solving - age 16, Mafhemafics confenf - age 13, 
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Mathematics content - age 16. The analytic approach used was multi- 
level modeling with the program Hierarchical Linear Modeling (HLM5; 
Raudenbush, Bryk, Cheong, & Congdon, 2000). Student mathematics 
achievement as measured by the SAIP tests was the level-1 outcome 
measure and student traits derived from the Student Questiormaire 
(CMEC, 2005) were the Level-1 explanatory variables. The intercept and 
gradients of fhe Level-1 model were then modeled with the school 
variables (Level-2) that were derived from a principal componenfs 
analysis of a selecfion of ifems from fhe School Questiormaire and from 
fhe aggregation of facfor scores derived from fhe Sfudenf Questiormaire. 

Analysis of the Student Questionnaire 

Analyses were conducfed on fhe 119 ifems of fhe sfudenf questiormaire 
fo reduce fhe number of variables af Level-1 info a small sef of 
meaningful facfors fo make fhe analysis of fhe sfudenf dafa more 
manageable. Lurfher, fhe sfudenf facfor scores were aggregafed fo form 
school means, producing school-level indicafors relafed fo insfrucfional 
practices, resources, and sfudenf affifude. 

A sef of 32 ifems was selecfed from fhe SAIP Mafhemafics Sfudenf 
Questiormaire. We used fhe following criferia: fhe included ifems were 
relevanf and accessible fo policy making, had relafively good response 
rafes (-10% or less missing dafa per ifem) fo minimize fhe effecfs of 
missing dafa, and had relafively high communalifies for principal 
componenf analysis. Principal componenf analysis followed by a 
varimax rofafion of fhe componenfs wifh eigenvalues greafer fhan or 
equal fo one (Mulaik, 1972) yielded a five-facfor solution accounting for 
43.3 per cenf of fofal variance. The full sfudenf file (n = 43,314) was 
initially used for fhis analysis. However when initial analyses were run 
on fhe 32 quesfiormaire ifems of inferesf, only 27,677 sfudenfs (63.9%) 
had complefe dafa. The missing dafa was nof sysfemafic in fhaf fhe 
majority students had complete data records, or at most one or two 
missing responses to questiormaire items. However, when collated for 
calculafions of correlations, incomplefe cases were delefed resulting in 
fhe subsfanfial overall decrease in complefe records. To address fhis 
problem an EM estimation procedure for missing dafa impufafion 
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(Systat, 2002) was used to estimate and impute all missing data. The final 
five-factor solution is reported in Table 2. 

Table 2: Five Factor Solution: Student Questionnaire 


Factor Label 

Item stem (paraphrased) 


Questionnaire Loading 
item # 


Student beliefs about mathematics 

(l=Strongly disagree . . . 5=Strongly agree) 


Math is more difficult than other school subjects 

Q14A 

-0.516 

I am not very interested in mathematics 

Q14B 

-0.717 

I learn lots of new things in mathematics 

Q14C 

0.574 

Math is an important school subject 

Q14D 

0.738 

Math is important for my future studies 

Q14E 

0.784 

Many good jobs require mathematics 

Q14F 

0.666 

Instructional supports used by students 



(How often ? 1-Rarely or never 5= Almost every day) 


You & your parents work on mathematics homework 

Q20A 

0.439 

You & your parents work on other homework 

Q20B 

0.419 

In math course we work in pairs or small groups 

Q22E 

0.386 

In math we use math books & magazines 

Q23A 

0.335 

In math we had guest speakers or experts 

Q23B 

0.672 

In math we use computers 

Q23C 

0.750 

In math we use the internet 

Q23J 

0.771 

In math we use the computer lab 

Q23K 

0.745 

Instructional practices 



(How often ? 1-Rarely or never 5= Almost every day) 


In math courses this year. . . 



The teacher gives notes 

Q22A 

0.428 

The teacher shows us how to do problems 

Q22B 

0.622 

We participate in math projects 

Q22C 

0.391 

We are taught different ways to solve problems 

Q22D 

0.587 

The teacher assigns homework 

Q22G 

0.479 

We discuss quiz or tests 

Q22H 

0.513 

We work alone on assigned work 

Q22I 

0.335 
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We work on exercises from textbook 

Q22J 

0.540 

We study the textbook 

Q22K 

0.514 

The teacher reads from the textbook 

Q22L 

0.548 

Teachers asks questions of students 

Q22M 

0.613 

Students ask teacher questions 

Q22N 

0.499 

Causes of mathematics performance 

(l=Strongly disagree . . . 5=Strongly agree) 

To do well in math you need hard work 

Q16A3 

0.600 

To do well in math you need encouragement - teachers 

Q16A4 

0.789 

To do well in math you need encouragement - parents 

Q16A5 

0.771 

To do well in math you need good teaching 

Q16A6 

0.553 

Disciplinary climate 

{How often? 1 -Rarely or never 5= Almost every day) 


In mathematics courses this year. . . 

There is noise or disorder in the classroom 

Q220 

0.792 

We lose 5-10 minutes because of disruptions 

Q22P 

0.784 


The first factor was labeled Student Beliefs about Mathematics and 
consisted of six items related to the extent to which students viewed 
mathematics as useful and important in their lives, and the extent to 
which the students found mathematics difficult or interesting. High 
scores on this factor reflect positive student perceptions in the sense of 
viewing mathematics as useful, interesting, and not difficult. In contrast 
low scores reflect negative views. 

The second factor. Instructional Supports Used by Students, contained 
eight items indicating the extent to which the students used parental 
help with mathematics and other homework, and the extent to which 
computers, mathematics literature, and mathematics experts were part of 
mathematics instruction in their classroom. This factor could be viewed 
as an index of student use of non-teacher instructional support. High 
scores on this factor reflect high frequency of student use of these 
supports, while low scores indicate little or no use of these supports. 

The third factor. Instructional Practice, consisted of 12 items that dealt 
with the frequency of various instructional activities in the mathematics 
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classroom such as working on mathematics projects, asking questions, or 
working on problem solving. High scores on this factor correspond to a 
greater frequency of use of a variefy of insfrucfional pracfices in fhe 
mafhemafics classroom suggesfing a high diversify in classroom 
insfrucfional practices. Low scores on this factor could mean that 
students perceived less frequent use of fhese practices. Low scores also 
suggest a more simplified insfrucfional environmenf. 

The fourfh factor. Causes of Mathematics Performance, consisted of four 
ifems related fo fhe affribufions sfudenfs make to mathematics 
achievement such as working hard, receiving encouragement from 
feachers and parenfs, or receiving good teaching. A high score on fhis 
facfor means fhaf sfudenfs express a high level of accepfing fhe belief 
fhaf hard work, good insfrucfion, and encouragemenf (from bofh teacher 
and parenf) enhance mafhemafics performance. 

The fiffh facfor. Disciplinary Climate, consisted of fhe fwo ifems on fhe 
sfudenf quesfiormaire fhaf dealf wifh fhe frequency of disrupfions and 
noise in fhe mafhemafics class. Sfudenfs wifh high scores on fhis facfor 
reported fhaf fhere were high levels of disrupfion and noise in fheir 
mafhemafics classrooms. A low score on fhis factor would reflect 
students perception of low levels of classroom disrupfion and noise. 

Facfor scores were calculated for each sfudenf and appended fo each 
sfudenf's record fo serve as sfudenf-level correlates of learning. Furfher, 
fhe sfudenf scores on each facfor were aggregafed info school averages 
fhaf were evenfually appended onto each school record for fhe 
subsequenf HLM analysis. These aggregafed facfor scores were used as 
indices of school fraifs albeif based upon sfudenf percepfions. 

School Questionnaires 

School quesfiormaires were complefed by fhe school principal in fhe 
same fime period as fhe adminisfrafion of fhe SAIP mafhemafics fesfs fo 
sfudenfs. A fofal of 1,681 school quesfiormaires were included in fhe 
dafasef. The school quesfiormaires consisfed of a fofal of 256 ifems. To 
make fhe dafa sef more manageable for sfafisfical modeling, a reduced 
sef of 27 ifems was selecfed using fhe same criteria used fo idenfify fhe 
ifems in fhe sfudenf quesfionnaire fo be included in fhe analyses: 
relevance, accessibilify, good response rafes, and communalifies. 
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The principal components analysis of these 27 items resulted in a 
five-factor orthogonal solution (Table 3) that accounted for 47.7 per cent 
of total variance. 

The first factor. Limitations to Learning, consisted of eight items 
asking principals to estimate the proportion of single-parent families and 
students with problems. They also estimated the extent to which various 
school characteristics limited the capacity of their school to provide 
effective instruction: concerns about student ability, lack of parental 
support, community conditions, or shortage of mathematics teachers. A 
high score on this factor means that a principal perceives that student, 
home, and community characteristics constrain instructional 
effectiveness of the school in mathematics. A lower score would be 
related to principals who view the student and home traits of the school 
to be less constraining, perhaps even positive elements in enhancing 
school effectiveness. 

The second factor. Climate, consisted of four items related to staff 
morale, school spirit, staff and student pride in school, and community 
support. A high score on this factor would indicate that the principal 
views staff, student, and community support and spirit for the school to 
be positive, and a low score would indicate negative perceptions of spirit 
and support. 

The third factor. Parents, consisted of eight items indicating the 
extent of parental involvement in school activities such as curriculum 
committee participation, classroom volunteer participation, fund raising, 
or interacting with staff. A high score on this factor would mean that the 
principal views parental involvement with the school as high and 
positive in the many areas of school activities and operations. 

The fourth factor. Student Status, consisted of four items related to 
the student characteristics in the school: the number of full-time 
students, the proportion of students living close to the school, and their 
access to computers. This factor could be considered an index of school 
size and proximity to students' home community. A high score would 
mean that the principal perceives the school to be of large size with a 
good level of computing resources, and with most, if not all, students 
living in the school catchment area. 
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Table 3: Five Factor Solution: School Questionnaire 

Factor Label 

Item stem (paraphrased) Questionnaire Loading 

item # 


Limitations to Learning (What percentage? 1= <,10%.. .3= >.25%) 

% of students with learning problems needing special attention Q07D 0.623 

% of students from single parent families Q07E 0.485 

% of students with health problems that inhibit learning Q07F 0.684 

(Asked to estimate extent of limiting capacity 1= None ... 4 = A lot) 

School instructional capacity limited by lack of parental support Q16A 0.688 

School instructional capacity limited by student abilities Q16B 0.735 

School instructional capacity limited by student's home background Q16C 0.821 

School instructional capacity limited by community conditions Q16D 0.744 

Instruction limited by shortage of math teachers Q17B 0.333 

Climate (l=Strongly disagree ... 4= Strongly agree) 

This school is supported by the community Q31G -0.601 

Staff morale is high in this school Q31H -0.771 

There is a strong school spirit Q31I -0.821 

Students and staff take pride in this school Q31J -0.787 

Parents (l=None ... 4= A lot) 

Extent that parents act as volunteers in classrooms Q28A 0.391 

Extent that parents volunteer in monitoring student behaviour Q28B 0.404 

Extent that parents serve on committees on curriculum Q28C 0.675 

Extent that parents serve on committees on finance or administration Q28D 0.671 
Extent that parents influence selection of principal or teachers Q28E 0.555 

Extent that parents serve on committees on student conduct Q28F 0.640 

Extent that parents interact with staff on matters of their own child Q28G 0.358 

Extent that parents help raise funds for school Q28H 0.435 

Student Status (l=low . . . 4=high) 

Full-time equivalent students in school Q05 0.776 

Percentage of students within walking distance of school Q07A 0.394 

% of students using subsidized transportation Q07B -0.369 

How many computers in school & available to students? Q18 0.773 

Factors of Student Achievement (l=Strongly disagree ... 4= Strongly agree) 

Students can achieve if they work hard Q31B 0.655 

Students can achieve if they are taught well Q31C 0.679 

Students can achieve with adaptations for special needs Q31D 0.525 
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The last factor. Factors of Student Achievement, consisted of fhree ifems 
asking principals abouf perceptions of correlafes of sfudenf achievemenf: 
efforf, good insfrucfion, and adapfafions for special needs. A high score 
on fhis factor means that the school principal views hard work by 
students, effective teaching, and school adaptations as major 
contributors to student achievement in mathematics. 

Factor scores on each of fhe five factors were calculafed for each 
school in fhe dafa sef and appended to fhe school's record. This resulfed 
in a sef of 10 variables af level-2: five derived from fhe aggregafed 
sfudenf facfors and five from fhe school questionnaire. 

MODELING STUDENT ACHIEVEMENT IN MATHEMATICS 

A hierarchical linear modeling approach (Raudenbush & Bryk, 2002) was 
used in fhese analyses. The oufcome measures as developed by fhe 
CMEC were fhe Confenf and Problem Solving fesfs of fhe SAIP 
mafhemafics assessmenf. The sfudenf-level correlafes for fhe initial sef of 
analyses reporfed in fhis paper consisted of fhe five facfors scores af fhe 
sfudenf level (Table 2) and sfudenf gender (males coded 0 and females 
coded 1). The 10 school-level correlafes were fhe five facfors derived 
from fhe school questionnaires items (Table 3), and the aggregated five 
facfor scores derived from fhe sfudenf quesfionnaire. As indicafed 
earlier, fhe dafa were divided info four sefs: by age (13 and 16 year-olds) 
and mafhemafics domain (Confenf and Problem Solving), resulting in 
four HEM models. 

Explained Variance 

The firsf sfep in fhe analyses was to evaluafe fhe proporfion of fofal 
variance in mafhemafics achievemenf scores fhaf was accounted for by 
school level variables: fhe infra-class correlation or p (Raudenbush & 
Bryk, 2002). The proporfion of variance in achievemenf scores in bofh 
domains affribufable fo schools (Table 4) was relatively tow (<.20). There 
were slighf differences across age groups wifh 13 year-old sfudenfs 
having greafer proporfion of variance accounfed for by schools (0.18 and 
0.19) fhan fhaf of fhe 16 year-old sfudenfs (0.15 and 0.15). 
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Table 4: The proportion of variance in student achievement scores accounted for by 

schools 

Test 

P 

Problem solving - age 13 

0.18 

Problem solving - age 16 

0.15 

Math content - age 13 

0.19 

Math content - age 16 

0.15 


Level-1 - Student Correlates of Mathematics Achievement. Next the 
student level predictors of mathematics achievement were modeled 
(Table 5). Two student factors were related to mathematics achievement 
for bofh ages and domains. The ofher fhree facfors had weaker 
relafionships fhaf varied across age and domain. 

For all four dafasefs. Student Beliefs about Mathematics was 
significanfly relafed fo mafhemafics achievemenf (>0.30). As posifive 
views on fhe value of and interesf in mafhemafics increased, 
mafhemafics achievemenf increased for sfudenfs in bofh age groups and 
for bofh mafhemafics domains. The direcfion of influence is open fo 
conjecfure because posifive views abouf mafhemafics could enhance 
inferesf and fherefore achievemenf, buf if could also be argued fhaf high 
achievemenf sfimulafes posifive views abouf mafhemafics. Likewise 
Instructional Supports used by Students was significanfly relafed fo 
mafhemafics achievemenf for all four dafa sefs, buf in an inverse maimer 
(<-0.15). As reporfed use of insfrucfional supporf increased, mafhemafics 
achievemenf fended fo decrease. This may appear fo be somewhaf 
counferinfuifive because parental assistance with homework is often 
thought to lead to an increase in student achievement. The response 
distributions for fhese ifems is highly skewed showing fhaf fhe majority 
of sfudenfs reporfed liffle or no use of any of fhe lisfed supporfs fhaf may 
have influenced fhe nafure of fhis relafionship. However, because fhis is 
a sfudenf-reporfed percepfion, if may be fhaf fhe sfudenfs reporfing 
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higher levels of use of supporfs fend to be weaker students, and this is 
consistent with other reported research (Ho & Willms, 1996). 

Table 5: Student and School Level Coefficients 


Correlate CONTENT PROBLEM 



13 

16 

13 

16 

Student-level 





Student math beliefs 

.36 

.33 

.38 

.37 

Instructional supports 

-.18 

-.22 

-.22 

-.29 

Instructional practices 

.03 

.08 

.04 

.10 

Causes of math 

-.08 

0 

-.06 

0 

Discipline climate 

0 

0 

0 

0 

Gender 

0 

-.09 

.10 

.07 

School-level correlates to intercept 




Limits to learning 

-.14 

-.21 

-.14 

-.18 

Instructional supports 

-.12 

-.19 

-.10 

-.19 

Causes of math 

0 

-.22 

0 

-.22 

Discipline climate 

0 

0 

0 

-.17 

Student math beliefs 

.13 

0 

.11 

0 

School climate 

0 

0 

-.04 

-.05 

Parent engagement 

0 

.05 

0 

.06 

Student status 

.07 

.06 

.05 

0 

Student achievement 

.05 

0 

.05 

0 

Instructional practices 

.08 

0 

0 

0 

a. Only coefficients significant at the a 

= 0.05 level are 

reported, otherwise 0. 



Instructional Practices was positively but weakly related to 
mathematics achievement for both age groups and domains of 
mathematics achievement. This result indicates that an increased 
diversity of classroom practices is associated with higher levels of 
mathematics achievement. A weak negative relationship (<-0.10) 
occurred between Causes of Mathematics Performance and mathematics 
achievement in both domains for 13 year-old students. As the strength of 
belief in these causes (both internal and external) increased, mathematics 
achievement for 13 year-olds tended to decrease. In contrast, no 
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significant relationship occurred between Causes of Mathematics 
Performance and achievement in the Problem mathematics domain for 16 
year-old students. 

Perceptions of classroom disorder and disruption in the mathematics 
classroom were not significantly related to mathematics achievement. 
Because this factor is based upon student perception elicited in two 
questiormaire items, it may well be that students are not particularly 
sensitive to evaluating disciplinary climate when based upon time 
estimates of classroom activities (Ma & Willms, 2004). More objective 
measures of classroom order and organization may shed light on the 
accuracy and consistency of student perceptions of classroom order and 
noise. 

The relationships between student gender and mathematics 
achievement were weak and mixed. In the mathematics Content domain, 
16 year-old males tended to do better than females although there was 
no significant gradient for 13 year-old students. However, in the 
mathematics Problem domain, females tended to have higher 
achievement than males in both age groups. 

Level-2 - The Intercept: Mean School Mathematics Scores. The mean 
school mathematics scores (the intercept terms in the Level-1 models) 
were modeled using the 10 school-level correlates (Table 5). Even after 
conditioning on the school correlates, a significant variation occurred 
between schools in terms of the mean school mathematics achievement 
scores for both age groups and for both mathematics domains. This 
variation suggests that additional correlates of learning account for 
differences in mathematics achievement of students in the school that 
have not been included in these data sets. 

Limitations to Learning and Instructional Supports were significant 
correlates of average school achievement in both mathematics domains 
and for both age groups. With the Limits to Learning factor, as principals 
perceived that limitations increase, the school mean mathematics scores 
tend to decrease. Not only was the direction of this relationship the same 
(negative) for all four datasets, the coefficients were very similar for each 
age. To an extent this negative relationship may be a self-fulfilling 
prophecy. Schools in which the principal views the students, their 
homes, and communities as problematic, thereby imposing limits on 
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what the school can do, tend to have lower achievement. In contrast, 
schools in which the principal views students and their homes and 
communities in a more positive light - with these elements not creating 
impediments to student learning - tend to be associated with better 
overall achievement. Alternatively, in schools with weak mathematics 
achievement the principal tends to develop a negative view, while in 
schools with strong mathematics achievement the principal tends to 
develop a positive view. It should be noted that these perceptions would 
likely be rather global impressions, for example, responses to the item 
related to the perceptions regarding single parent families are likely 
based on general impression rafher fhan empirical evidence. 

Similar fo fhe resulfs for Level-1, fhe Instructional Supports facfor 
(averaged across sfudenfs wifhin each school) was negatively relafed fo 
school mafhemafics performance (-0.10 fo -0.19). This resulf means fhaf 
as fhe average reporfed use of insfrucfional supporfs wifhin fhe school 
increases, fhe mean mafhemafics scores fend fo decrease. Schools in 
which sfudenfs reporf less use of insfrucfional supporfs tend to have 
higher mathematics scores on average. 

Causes of Mathematics was negatively related to mathematics 
performance for bofh Confenf and Problem Solving for fhe 16 year-old 
sfudenfs (-0.22), buf fhere was no signiticanf relafionship for fhe 13 year- 
old sfudenfs. This resulf can be inferprefed fo mean fhaf as 16 year-old 
sfudenfs in fhe school increased fheir attribution of mafhemafics 
achievemenf fo hard work and feacher and parenf encouragemenf, fhe 
mean mafhemafics score for fhe school fended fo go down. To an exfenf 
fhis relafionship could be viewed as affribufing performance in 
mafhemafics fo exfemal elemenfs in fhe sfudenfs' world: feachers and 
parenfs. However fhe inclusion of hard work in fhis facfor, an infernal 
elemenf, is somewhaf counferinfuifive. This facfor will need to be further 
investigated to better understand this relationship. 

Disciplinary Climate is significantly predictive only for 16 year-old 
sfudenfs in fhe Problem domain (-0.17). Here, as disciplinary climafe as 
reporfed by sfudenfs worsens (more disruption in mafhemafics classes, 
and higher scores on fhis facfor) fhe mean school Problem achievemenf 
scores fend fo decrease, a finding fhaf is consisfenf wifh expecfafion. The 
absence of fhis relafionship in fhe Confenf domain for bofh age groups. 
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and for 13 year-old students in the Problem domain, requires further 
investigation. 

Student Mathematics Beliefs were positively related to average 
mathematics scores only for 13 year-old students. As the positive views 
about mathematics of students in the school increased, there tended to be 
a increase in the mathematics achievement of the students. This 
relationship does not seem to be present in the perceptions of 16 year-old 
students. 

Level-2 - Modeling the Gradients. Gradients within the context of 
HLM5 (Raudenbush, Bryk, Cheong, & Congdon, 2000) are indices of the 
strength and direction of relationship between the student level 
correlates (e.g.. Mathematics Beliefs or Gender) and the outcome measure 
(mathematics scores). The student gradients expressed as coefficients of 
the potential explanatory variables at Level-1 (Table 5) were modeled at 
the school level with Level-2 correlates. 

The results for 13 year-olds in the level-2 model are shown in Table 
6. Mathematics achievement in the Content and Problem domains for the 
13 year-old students was influenced by their Beliefs about Mathematics 
(0.36 and 0.38) and their Use of Instructional Supports (-0.18 and -0.18). 
However, for the Content domain (Table 6) there were no significant 
Level-2 school correlates related to these two student gradients. For the 
Problem domain, the significant positive relationship between Student 
Beliefs about Mathematics and mathematics achievement (0.38) was 
influenced by two school-level correlates: aggregated or school average 
Student Beliefs about Mathematics (0.07) and by Instructional Practices (0.07). 
Hence for Problem Solving, the relationship between student beliefs 
about mathematics and achievement was not consistent across schools. 
Those schools with higher average scores on Student Beliefs about 
Mathematics and diversity of Instructional Practices had modestly higher 
Level-1 gradients between individual Student Belief about Mathematics and 
achievement. All other student gradients (Instructional Supports, 
Instructional practices and Gender) were essentially constant across schools 
and had no significant relationships to school-level correlates. 
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Table 6: School Level (Level-!) Coefficients: The Gradients 


Student Variable 
Gradient (13yr/16yr) 


School Correlates 


13 year old 16 year old 


Content Domain 

Student math beliefs (.36/33)^' 
Instructional supports ( -.18/-.22)‘ 

Instructional practices (.03/.08)® 
Causes of math ( -.08/-.04)® 
Gender (0/-.09) 



ns 

ns 

Limits(-.03) 


Support(.15) 


Student Ach(.04) 

ns 

ns 

ns 

ns 

ns 

ns 


Problem Solving 


Student math beliefs (.38/.37)'’ 

Student Beliefs(.07) 

Parents(-.03) 


Instruction Practices(.07) 

Discipline(-.09) 

Instructional supports (-.22/-.29)“ 

ns 

ns 

Instructional practices (.OT/.IO)® 

ns 

ns 

Gender (.10/.07)‘> 

ns 

ns 


a. No significant variation (x) between schools on this gradient 

b. Significant variation (x) between schools on this gradient 

c. No significant school-level correlates 


For 16 year-old students in the Content domain (Table 6), only the 
Instructional Supports gradient varied significantly across schools. As the 
average level of sfudenf Use of Instructional Support wifhin a school (fhe 
Level-2 correlafe - fhe school average) increases, fhe influence of sfudent 
use of instructional supporfs (fhe Level-1 gradienf) flattens. In ofher words 
fhe relafionship of sfudenf use of supporfs fo mafhemafics achievemenf 
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which is negative (-.22) is reduced. In addition, two other school 
correlates are significantly related to this gradient: Perceived Limitations 
to Learning (-.03) and Factors of Student Achievement (.04). However, the 
values are quite small, suggesting that the relationship of school 
correlafes fo sfudenf -level correlafes is weak. 

In fhe Problem domain, only one Level-1 gradienf. Student Beliefs 
about Mathematics, varied significanfly across fhe schools in fhe 16 year- 
old sfudenf sample (Table 6). And fhis sfudenf correlafe is negafively 
influenced by average parenfal involvemenf in schools (Parents) and by 
fhe average level of disruption and noise in fhe mafhemafics classrooms 
(Discipline). However, fhe values (-0.03 and -0.09, respectively) are small, 
suggesting that the relationship to student performance as measured by 
mafhemafics achievemenf is slighf. The relafionship of average sfudenf 
percepfion of classroom disruption (Discipline) cerfainly is in line wifh 
expecfafions: increased disruption relafed fo a decrease in fhe posifive 
relafionship of Student Beliefs about Mathematics on mafhemafics 
achievemenf. However, fhe weak negafive relafionship of average 
reporfed levels of parenfal involvemenf wifh school affairs (Parents) is 
counferinfuifive and requires furfher invesfigafion. 

DISCUSSION 

The resulfs reporfed in fhis arficle reveal some consisfenf sfrong 
relationships af fhe sfudenf level and an imporfanf characteristic of 
large-scale assessmenf dafa in Canada. Sfudenfs' beliefs abouf 
mafhemafics are posifively relafed fo mafhemafics achievemenf as 
measured by fhe SAIP fesf of Confenf and Problem Solving for bofh age 
groups and for bofh domains of mafhemafics. Also fhe percepfions of 
school principals of fhe school environmenf (Limitations to Learning) were 
relafed fo overall school performance. We see fhaf fhere is a relafionship 
befween percepfions or affifudes and mafhemafics achievemenf af bofh 
fhe sfudenf and school levels. The policy relevance here is clear: 
enhancing sfudenf and principal percepfions and beliefs could enhance 
sfudenf achievemenf and fherefore school performance. 

Second and somewhaf counferinfuifively, as sfudenfs reporf fhaf fhe 
use of insfrucfional supporfs (such as parenfal assisfance wifh 
mafhemafics homework and fhe use of compufers in fhe mafhemafics 
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classroom) increases, there is an associated decrease in mathematics 
scores. The policy relevance here is unclear because to reduce parental 
involvement with a students' work at home on school subjects would not 
be accepted by most educators and parents. As indicated earlier, 
inspection of the response distributions for fhe ifems relafed fo 
insfrucfional supporf revealed fhaf fhe majority of sfudenfs reporfed 
liffle or no use of any of fhe supporfs lisfed in fhe sfudenf quesfiormaire. 
If may be fhaf fhe sfudenfs who did reporf fhey made use of supporf 
were low performing sfudenfs and fhus fhe negafive relafionship. If may 
also be fhaf fhis factor is an index of sfudenf independence: fhose 
sfudenfs reporfing less use of parenfal assisfance are relatively 
independenf in fheir work on mafhemafics and fhis is associafed wifh 
higher levels of performance. However, from a research perspecfive fhe 
direction is clear, fhe need exisfs for collecfing more direcf indicafors of 
insfrucfional supporf use fo better undersfand ifs function in relafion fo 
sfudenf achievemenf in a general manner. If will be of inferesf fo hack 
fhese pafferns fhrough similar fufure analyses conducfed on subsefs of 
fhe dafa for differenf educafional jurisdicfions. 

A pattern fhaf emerged in fhese analyses of SAIP Mafhemafics 2001 
was fhe general sfabilify of fhese sfudenf-level relationships (fhe 
gradienfs) across schools. Mosf of fhe sfudenf-level coefficienfs did nof 
show significanf school variation and, as a consequence, many did nof 
have any significanf school-level correlates influencing fheir values, 
suggesting fhaf fhe correlates have a common influence in schools 
nationally. 

In regard to large-scale assessment data, the analysis of fhe four 
dafasefs from SAIP Mafhemafics 2001 showed fhaf less fhan 20 per cenf 
of variation in achievemenf measures was affribufable fo schools. This is 
a characteristic and significanf feafure of large-scale assessmenf dafa in 
Canada (OECD, 2001; Anderson, 2002): fhe relatively low proportion of 
variation in sfudenf achievemenf fhaf can be affribufed fo schools 
(<20%). An infemafional average of OECD counfries as assessed in fhe 
Programme of International Student Assessment (PISA) is .35 (OECD, 2001). 
Alfhough in Canada school differences are modesf, fhe influence of fhese 
differences on overall sfudenf achievemenf should cerfainly be 
considered by fhose who publish school rankings (e.g., Cowley, Easton, 
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& Walker, 2001) because these rankings imply that most if not all 
variation in averaged school results, regardless of how fhey are derived, 
are affribufable fo school-based facfors. The evidence does nof supporf 
fhis implicafion. 

In fhe resulfs reporfed here if is clearly evidenf fhaf mean school 
performance did vary significanfly across schools, and fhese scores were 
relafed fo school characferisfics, in particular the extent to which 
principals reported perceiving student, home, and community traits as 
limiting elements of school effectiveness, and fhe average reporfed 
sfudenf use of insfructional supporfs. This firsf relafionship does appear 
fo be accessible fo policy infervenfion: principals' beliefs could be 
influenced by communications sfrafegies in fhe school sysfem. The 
second relafionship is similar fo fhaf found af fhe sfudenf level for fhe 
use of insfrucfional supporf: fhe more use of or dependence upon 
insfructional supporf reporfed by sfudenfs, on average wifhin fhe school, 
fhe lower fhe associafed mafhemafics scores. There is clearly a 
relafionship here, buf if would be imporfanf fo collecf furfher dafa on 
fhese variables fo ensure fhaf whaf sfudenfs are reporting is indeed an 
accurafe reflecfion of fhe exfenf fo which fhey do homework and fhe 
exfenf fo which parenfs assisf fhem, and fo collecf direcf evidence of fhe 
insfrucfional practices used in mafhemafics such as compufer use, fhe 
use of reference maferials, or resource people for mafhemafics classes. 
This should help fo beffer undersfand fhese relafionships and fheir 
ufilify in policy formulafion and implemenfafion. 

This infroduces an importanf direction these analyses identify in 
evaluating fhe ufilify of fhe SAIP dafasefs for sfafisfical modeling of 
school performance: fhe need for more and befter dafa on sfudenf, 
school, class, and home variables, an issue more fully discussed in 
Rogers ef al. (2006). The negative relafionship befween use of 
insfrucfional supporfs fo mafhemafics achievemenf, fhe non-significanf 
relafionship befween classroom disruption and noise fo mafhemafics 
achievemenf, and fhe sfarfling sfabilify of sfudenf gradienfs across 
schools all suggesf fhe need for furfher invesfigafion wifh beffer, more 
direcf measures of correlafes of learning oufcomes and fhe developmenf 
of insfrumenfafion fo collecf fhis informafion. An obvious need exisfs fo 
collecf dafa af fhe class or feacher level because less fhan 20 per cenf of 
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variance in student performance can be attributed to schools, meaning 
that over 80 per cent can be attributed to students and classes. The 
analyses reported here were based upon the results generated by SAIP 
Mathematics 2001, in effect statistical modeling based upon found dafa. 
The findings of sfudies such as fhis should be used fo inform fhe shape 
of fufure assessmenf programs. Because dafa such as fhese are collecfed 
annually in provincial and nafional assessmenf programs, wifh 
appropriafe planning and design, dafa collecfion sfrafegies could be 
modified wifh liffle or no addifional expendifure fo collecf dafa fo beffer 
address policy- and pracfice-relevanf evidence requiremenfs. 

NOTES 


^ The COLO project is funded by the Canadian Educational Statistics 
Council and the Social Sciences and Humanities Research Council. 
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